Repository navigation
fix(examples): example 39 must run, and must name the error it really raises - #843
Merged
NandhaKishorM merged 3 commits intoOct 2, 2026
Merged
Conversation
…r it really raises `os.path.dirname(MODELS["english"])` handed `os.path` a `Router` spec: `_common.checkpoint` returns `(repo, subfolder)` when `models/` is not in the checkout, which is every checkout, since `models/` is not in the repository. The file died with `TypeError: expected str, bytes or os.PathLike object, not tuple` at case 2, so cases 3, 4 and 5 never ran under a docstring that says every case is executed. The path now comes from `LOCAL_MODELS`, which is a directory by contract. Item 3 promised "a low-level RuntimeError" for a `choice` question with empty criteria, three lines above the output that prints `ValueError`, and its conclusion sent both shapes to the decision head: "reach the scorer with zero options and blow up in top-k". `_check_question` rejects them before the state is encoded, naming the question -- measured with the English checkpoint and with none.
Run the page on the tiny checkpoint this suite already builds, and let its own output be the witness: for each numbered case, the exception types the banner lists must be exactly the types that case's section prints. Nothing in CI executes examples, and no suite read this page's prose, so both defects above were invisible. Also pins the guard the page quotes to the comparison in `laya/agent.py` (by the name `render_options` is stored under, so a rename survives), its option-count sentences to the `range()` bounds it sets up, its missing-path case to `LOCAL_MODELS`, and no `os.path` call to a `Router` spec -- with `_common.checkpoint` witnessed returning a non-path when `models/` is absent. The two sentences this replaced are kept verbatim as controls, and three arms (renamed guard variable, collapsed print, reworded message) are what the gate must survive.
Found by mutation, not by reading: the `os.path` walk matched `func.value.id == "os.path"`, which
is never true for `os.path.join(...)` -- the callee is `Attribute(value=Attribute(Name('os')))`. The
rule that is supposed to refuse a `Router` spec to a filesystem call therefore saw no call at all,
and the arm that reintroduces the crash only went three of four checks red. A dotted-name walk fixes
it, with a non-vacuity check that at least one `os.path` call is visible.
The option-count rule read a space-joined copy of the page, so a sentence wrapped across two `print`
calls never matched `(\d+) options` -- the count it was policing was the empty set, and an arm that
moves 120 to 110 survived. Whitespace is now collapsed before the prose is searched, and the rule
asserts the prose states a count rather than quietly passing on none.
4 tasks done
Owner
|
Merged for 0.3.24. Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, Thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/39_error_handling.pyhas two defects, and neither is visible from CI: examples are notexecuted by the test job, and no suite read this page's prose.
1. The file does not run. Its own docstring says "Every case below is executed and its real
behaviour reported". At
upstream/mainit stops at case 2:MODELSis theRouterspec dict, andexamples/_common.py'scheckpoint()returns the localdirectory only when
models/exists in the checkout -- otherwise(BUNDLE_REPO, subfolder).models/is not in this repository, so every clone that runs this example gets the tuple. Cases 3,4 and 5 -- the three that need an agent -- never ran anywhere.
The fix takes the path from
LOCAL_MODELS, which is a directory by contract:2. Item 3 describes an error that does not happen. The banner promised:
and under its own timings it printed:
Neither is true, and the page printed the contradiction two lines above it:
Agent._check_questionrefuses both shapes insidepredict_batchbefore the state is encoded --it never reaches
render_options, the decision head, ortop-k. Measured with the Englishcheckpoint, the traceback is
predict -> system_one -> predict_batch -> _check_question.Why
Case 3 is the shape a caller produces from a typo'd preset or an empty form field. A reader who
trusts the banner writes
except RuntimeError, theValueErrorescapes it, and the failure theyget is worse than the one documented. And the whole page -- the one an
AGENTS.mdpoints acontributor at for "the failure modes a caller should handle" -- exits 1 on a fresh clone, so
nobody finds out.
tests/test_load_errors.pyalready owns cases 2 and 4 of this page (sections "missing directory"and "a question whose options do not fit"), builds a loadable tiny checkpoint, and is in the CI
matrix. Section 7 spends that tiny agent on the page: it runs the example and lets the page's own
output be the witness. For each numbered case, the exception types the banner lists must be
exactly the types that case's section prints. Rules:
exec'd with_common.loadanswered by the tiny checkpoint built 40 lines above, sothe crash class is caught by execution, not by regex. Its output is sliced per heading; the banner
is outside those slices, so the banner cannot vouch for itself.
Routerspecs are witnessed as non-paths by calling_common.checkpoint("english")withhas_localstubbed off -- the state of every CI job -- rather than asserting "it's a tuple", so afuture str-shaped spec representation still passes while a real directory would not.
_check_question:ValueErroralone fits a hundred other raises.len(markers) != len(render_options(q))) is pinned to the comparisonin
laya/agent.pythrough a regex on the assignment, so the count variable can be renamed; thearm that renames
n_opts->opts_countstays green.range()bounds in its code, and each"N options"labelmust name the dict that actually has N entries -- an arm that pairs
"120 options"with the160-entry dict goes red, which is the trap of "fixing" this page by editing only the prose.
top-k,reach the scorer,blow up). The two sentences this PR replaced are kept verbatim inside the test as controls, so arule that matches nothing cannot pass for being empty.
How it was verified
upstream/main'stests/test_load_errors.pycounts 23 checks; here 49 (+26), exit 0. Nolaya/file is changed by this PR. The fixed page, real English checkpoint, MPS, torch 2.14.0:
Cases 4 and 5 read the same as before because they were only ever wrong in the banner's framing of
case 3 -- the crash meant nobody had seen them since
models/stopped being vendored.The gate was mutation-checked over 17 arms, each restoring every touched file from a byte snapshot
and requiring the suite's completion line (a crash or an aborted run scores ABORT, not a kill). All
17 arms:
M5 mutates
laya/agent.py(the raise becomesRuntimeError), so the gate is shown to be watchingthe runtime and not merely the words -- and the banner then disagrees with the page's own output.
M11 is the arm that closes the cheap way out: deleting case 2 also makes the file run.
The first harness run was 15 of 17, and both misses were my gate, which is commit 3:
os.pathwalk testedfunc.value.id == "os.path", which is never true foros.path.join(...)(the callee isAttribute(value=Attribute(Name('os')))), so the rule thatrefuses a
Routerspec to a filesystem call saw no call at all -- M1 went 3/4 instead of 4/4;printcalls never matched(\d+) options: the set it policed was empty and M10 survived.Commit 3 fixes both (
_dotted()walk, whitespace collapsed before the prose is searched) and adds anon-vacuity check to each: the
os.pathrule asserts that at least oneos.pathcall is visible,and the count rule asserts the prose states a count.
Checklist
mainFixes #
No issue: found by running the examples. No public API changed --
examples/39_error_handling.pyand
tests/test_load_errors.pyonly, and the new checks live in the suite CI already runs(
python tests/test_load_errors.py).